[Doc] database auto-update and database metrics by kaisecheng · Pull Request #187 · logstash-plugins/logstash-filter-geoip

kaisecheng · 2021-06-22T15:47:04Z

This PR adds more details on how database update behaves and explains how to use database metrics

kaisecheng · 2021-06-22T15:48:56Z

+
+==== Database Metrics
+
+You can monitor database status through https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#node-stats-api[Node Stats API]


please correct me on how to link the plugin doc to logstash doc

Suggested change

You can monitor database status through https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#node-stats-api[Node Stats API]

You can monitor database status through {logstash-ref}/node-stats-api.html#node-stats-api[Node Stats API]

kaisecheng · 2021-06-22T15:51:41Z

+The following request returns a JSON document containing database manager stats,
+including:
+
+* the database status and freshness
+** `geoip.database.*.status`
+*** `init` : initial CC database status
+*** `healthy` : using up-to-date EULA database
+*** `to_be_expired` : 25 days without calling service
+*** `expired` : 30 days without calling service
+** `fail_check_in_days` : number of days Logstash fails to call service since the last success
+* info about download successes and failures
+** `geoip.download.successes` number of successful checks and downloads
+** `geoip.download.failures` number of failed check or download
+** `geoip.download.status`
+*** `checking` : check and download at the moment
+*** `succeeded` : last download succeed
+*** `failed` : last download failed
+
+[source,js]
+--------------------------------------------------
+curl -XGET 'localhost:9600/_node/stats/geoip?pretty'
+--------------------------------------------------
+
+Example response:
+
+[source,js]
+--------------------------------------------------
+{
+  "geoip" : {
+    "database" : {
+      "ASN" : {
+        "status" : "healthy",
+        "fail_check_in_days" : 0,
+        "download_at": "2021-06-21T16:06:54+02:00"
+      },
+      "City" : {
+        "status" : "healthy",
+        "fail_check_in_days" : 0,
+        "download_at": "2021-06-21T16:06:54+02:00"
+      }
+    },
+    "download" : {
+      "successes" : 15,
+      "failures" : 1,
+      "last_check_at" : "2021-06-21T16:07:03+02:00",
+      "status" : "succeeded"
+    }
+  }
+}
+--------------------------------------------------
+


would like to know your opinion on whether this part should belong to Node State API or stay in plugin

geoip metrics only appear when user uses geoip plugin

Another good example. Thank you. ❤️
Your instincts are right. I believe this info provides the most user benefit here.
If you would like, we can set up a heading/section in the Node Stat API docs to expose this info. Rather than repeating the info, we can point them here. LMKWYT or if you'd like for me to set this up.

karenzone

More nice document with good examples, @kaisecheng. Thank you. ❤️
I left some comments in line and offered a suggestion for sharing this info with Node Stats. Let me know how I can help from here.

karenzone · 2021-06-22T19:34:01Z

 The Logstash open source distribution uses the MaxMind Creative Commons license
 database by default.

+==== Database Auto-update


Suggested change

==== Database Auto-update

[id="plugins-{type}s-{plugin}-database_auto"]

==== Database Auto-update

karenzone · 2021-06-22T19:45:32Z

+This plugin embedded with Creative Commons (CC) license databases.
+When Logstash works in air-gapped enviornment, CC license databases can be used indefinitely. 
+Logstash checks database update every day. It downloads the latest and can replace the old database on-the-fly.
+Once it downloads EULA license databases, it will not fallback to CC license databases.


Suggested change

This plugin embedded with Creative Commons (CC) license databases.

When Logstash works in air-gapped enviornment, CC license databases can be used indefinitely.

Logstash checks database update every day. It downloads the latest and can replace the old database on-the-fly.

Once it downloads EULA license databases, it will not fallback to CC license databases.

This plugin bundles Creative Commons (CC) license databases.

In air-gapped environments, Logstash can use CC license databases indefinitely.

Logstash checks for database updates every day. It downloads the latest and can replace the old database while the plugin is running.

After Logstash downloads EULA license databases, it will not fallback to CC license databases.

WRT "on-the-fly." I offered a suggestion, but I'm not sure if it's correct. Ideally, we should explain exactly how this works.

karenzone · 2021-06-22T19:48:51Z

+the geoip filter will stop enriching events in order to maintain compliance.
+Events will be tagged with `_geoip_expired_database` tag to facilitate the handling of this situation.
+
+NOTE: We recommend you allow Logstash to access internet to download databases


Suggested change

NOTE: We recommend you allow Logstash to access internet to download databases

Tip: When possible, allow Logstash to access the internet to download databases so that they are always up-to-date.

karenzone · 2021-06-22T19:49:55Z

+
+NOTE: We recommend you allow Logstash to access internet to download databases
+
+==== Database Metrics


Suggested change

==== Database Metrics

[id="plugins-{type}s-{plugin}-metrics"]

==== Database Metrics

karenzone · 2021-06-22T19:50:40Z

+
+==== Database Metrics
+
+You can monitor database status through https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#node-stats-api[Node Stats API]


Suggested change

You can monitor database status through https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#node-stats-api[Node Stats API]

You can monitor database status through {logstash-ref}/node-stats-api.html#node-stats-api[Node Stats API]

karenzone · 2021-06-22T19:55:08Z

+The following request returns a JSON document containing database manager stats,
+including:
+
+* the database status and freshness


Suggested change

* the database status and freshness

* database status and freshness

karenzone · 2021-06-22T19:58:27Z

+The following request returns a JSON document containing database manager stats,
+including:
+
+* the database status and freshness
+** `geoip.database.*.status`
+*** `init` : initial CC database status
+*** `healthy` : using up-to-date EULA database
+*** `to_be_expired` : 25 days without calling service
+*** `expired` : 30 days without calling service
+** `fail_check_in_days` : number of days Logstash fails to call service since the last success
+* info about download successes and failures
+** `geoip.download.successes` number of successful checks and downloads
+** `geoip.download.failures` number of failed check or download
+** `geoip.download.status`
+*** `checking` : check and download at the moment
+*** `succeeded` : last download succeed
+*** `failed` : last download failed
+
+[source,js]
+--------------------------------------------------
+curl -XGET 'localhost:9600/_node/stats/geoip?pretty'
+--------------------------------------------------
+
+Example response:
+
+[source,js]
+--------------------------------------------------
+{
+  "geoip" : {
+    "database" : {
+      "ASN" : {
+        "status" : "healthy",
+        "fail_check_in_days" : 0,
+        "download_at": "2021-06-21T16:06:54+02:00"
+      },
+      "City" : {
+        "status" : "healthy",
+        "fail_check_in_days" : 0,
+        "download_at": "2021-06-21T16:06:54+02:00"
+      }
+    },
+    "download" : {
+      "successes" : 15,
+      "failures" : 1,
+      "last_check_at" : "2021-06-21T16:07:03+02:00",
+      "status" : "succeeded"
+    }
+  }
+}
+--------------------------------------------------
+


Another good example. Thank you. ❤️
Your instincts are right. I believe this info provides the most user benefit here.
If you would like, we can set up a heading/section in the Node Stat API docs to expose this info. Rather than repeating the info, we can point them here. LMKWYT or if you'd like for me to set this up.

kaisecheng · 2021-06-22T21:02:31Z

@karenzone Setting up heading/section in the Node Stat API and pointing them to here sounds the right way. Because it avoids confusion of reading geoip metrics in Node Stat API doc but seeing nothing return when the pipeline doesn't involve geoip plugin.
Yes, it will be great if you can set up this 💪 Thank you

karenzone · 2021-06-22T23:11:14Z

@kaisecheng elastic/logstash#13019

karenzone

Very minor suggestions for your consideration. Otherwise, LGTM 🎉

karenzone · 2021-06-23T14:22:37Z

+the geoip filter will stop enriching events in order to maintain compliance.
+Events will be tagged with `_geoip_expired_database` tag to facilitate the handling of this situation.
+
+Tip: When possible, allow Logstash to access the internet to download databases so that they are always up-to-date.


Suggested change

Tip: When possible, allow Logstash to access the internet to download databases so that they are always up-to-date.

TIP: When possible, allow Logstash to access the internet to download databases so that they are always up-to-date.

All caps required to trigger the special "tip" formatting.

karenzone · 2021-06-23T14:23:49Z

+[id="plugins-{type}s-{plugin}-metrics"]
+==== Database Metrics
+
+You can monitor database status through {logstash-ref}/node-stats-api.html#node-stats-api[Node Stats API]


Suggested change

You can monitor database status through {logstash-ref}/node-stats-api.html#node-stats-api[Node Stats API]

You can monitor database status through the {logstash-ref}/node-stats-api.html#node-stats-api[Node Stats API].

Add documentation for database auto-update behavior and database metrics

e9f69e1

kaisecheng commented Jun 22, 2021

View reviewed changes

kaisecheng requested a review from karenzone June 22, 2021 15:56

roaksoax added the status:needs-review label Jun 22, 2021

karenzone reviewed Jun 22, 2021

View reviewed changes

fix doc

e90a1f7

karenzone mentioned this pull request Jun 22, 2021

Doc: Add geoip database API to node stats elastic/logstash#13019

Merged

add changelog

d0cad4f

karenzone approved these changes Jun 23, 2021

View reviewed changes

change metrics wording

f683de9

roaksoax added status:approved and removed status:needs-review labels Jun 23, 2021

kaisecheng merged commit 3f371f1 into logstash-plugins:master Jun 23, 2021

roaksoax removed the status:approved label Oct 1, 2021


		==== Database Metrics

		You can monitor database status through https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#node-stats-api[Node Stats API]

	==== Database Auto-update
	[id="plugins-{type}s-{plugin}-database_auto"]
	==== Database Auto-update

	NOTE: We recommend you allow Logstash to access internet to download databases
	Tip: When possible, allow Logstash to access the internet to download databases so that they are always up-to-date.

	==== Database Metrics
	[id="plugins-{type}s-{plugin}-metrics"]
	==== Database Metrics

	* the database status and freshness
	* database status and freshness

Conversation

kaisecheng commented Jun 22, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaisecheng Jun 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karenzone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaisecheng commented Jun 22, 2021

Uh oh!

karenzone commented Jun 22, 2021

Uh oh!

karenzone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kaisecheng Jun 22, 2021 •

edited

Loading